Lean Analytics - critical summary review

Inspired by Taiichi Ohno’s Toyota Production System and initiated by Yale alumni Eric Ries, the lean startup movement introduced a generation of entrepreneurs to the importance of quickly and accurately assessing progress through the Build–Measure–Learn loop.

“People are good at the build part,” said Lean Analytics co-author Benjamin Yoskovitz in a 2013 interview, the same year that the book was released. “They have an idea, they build something and then try to test it in the market. But it’s at this point where many people struggle.” Well, that’s where Lean Analytics should help them. Described as “a way of blowing out and digging into the ‘measure and learn’ aspects of lean startup,” the book is imagined as a sort of “dashboard for every stage of your business.” 

So, get ready to learn how to discern good from bad metrics, find the One Metric That Matters to you now, and learn how to figure out your stage of growth so that you can act accordingly.

Stop lying to yourself: embrace data

Even though we’re all delusional, entrepreneurs are probably the most delusional of all. You could even argue that, in a way, they have no choice but to live in a semi-delusional world if they want to survive: after all, a big part of their job is to convince others that something is true in the absence of good, hard evidence. However, even though small lies might be essential to their success in the long run, big lies inevitably lead to bubbles and epic failures. The only way to avoid them on time is by using data.

Data, being “the necessary counterweight to lying, the yin to the yang of hyperbole,” is a destroyer of delusions. Precisely because of this, it forms an essential part of the research-based lean startup methodology. One of the core concepts of this methodology is the build→measure→learn loop. This cycle – which highlights speed as a key ingredient of product development – suggests that the best way to meet your potential customers’ demands is by building a functional, imperfect product, share it with your users as such, find a way to measure their reactions, and then learn from these measurements to build a better iteration of the same product in the future. Within this incessant cycle, lean analytics focuses on the measuring stage. Obviously, the faster your organization manages to iterate through the cycle, the quicker you’ll be able to develop the proper product and find the right market for it. So, the better you measure things, the more likely you are to succeed.

What makes a good metric?

But how do you measure things better? How do you know which numbers are good and which are bad? Or, better yet, how do you know which numbers should drive you in the right direction and which might mislead you? 

According to Alistair Croll and Benjamin Yoskovitz, there are a few good rules of thumb that should help you discern between true and false metrics. A good metric, they write, is essentially one that has these three properties:

• A good metric is comparative. Only if a metric allows you to compare how things looked in the past as opposed to how they look now – that is to say, if it helps you understand which way things are moving – can it be considered a good metric. In other words, despite the absence of a number, the phrase “increased conversion from last week” is much more meaningful than “2% conversion.” 
• A good metric is understandable. If it’s difficult for people to remember and discuss it, then a change in the data will probably never translate to a change in the company culture – which is, after all, the point of measuring things.
• A good metric is a ratio or a rate. Most accountants need no more than a few ratios to understand, at a glance, the financial health of a company. There are at least three reasons why they, ratios, tend to be the best metrics available:
• Ratios are easier to act on. Distance traveled is merely informational, but speed – distance per hour – tells you immediately whether you need to go faster or slower to get to your destination on time.
• Ratios are inherently comparative. By definition, ratios encompass comparison and allow you to compare things over and over again. Compare your speed right now to that of your average speed during the last hour and you know whether you’re accelerating or slowing down.
• Ratios are also good for comparing factors that are somehow opposed, or for which there’s an inherent tension. You can divide the distance covered with your car by the number of traffic tickets you’ve earned and see if it makes financial sense to drive faster and break the speed limit.

Choosing the right metrics

By far the most important criterion to decide whether a metric is good or not is this seemingly simple question: what will you do differently based on changes in the metric in question? If the answer to this question reveals movement toward an objective of yours, then the metric is valuable. A good metric necessarily changes the way you behave. And it does so through one of these two ways:

• “Accounting” metrics, like daily sales revenue, can help you make better predictions and show you how close you are to an ideal model and whether your actual results are converging on your business plan.
• “Experimental” metrics, like the results of a test, can help you optimize the product, pricing, or market. Changes in these metrics should significantly alter your behavior. For example, if your tests reveal that a third of your respondents wouldn’t pay for a planned feature of your app, then it’s smart to not bother building it. Similarly, if your pink website generates more revenue than the blue one, then you must opt for pink, regardless of your personal preferences. As simple as that.

To sum up, if you measure something and that something is not attached to a previously determined goal (which might, in turn, affect your behavior), you are wasting your time. And if you don’t want to do this and instead want to choose the right metrics – you need to keep these five things in mind:

• Qualitative versus quantitative metrics. Just because something can’t be measured, it doesn’t mean that it isn’t a metric. Even so, qualitative metrics are “unstructured, anecdotal, revealing, and hard to aggregate;” quantitative, on the other hand, provide hard numbers but less insight.
• Vanity versus actionable metrics. Vanity metrics – such as the number of hits, page views, visits, followers and downloads – usually don’t change how you act: they just make you feel good. Actionable metrics help you pick a course of action and, thus, invite you to change your behavior.
• Exploratory versus reporting metrics. Reporting metrics keep you abreast of day-to-day operations, while exploratory metrics are speculative and deal with unknown insights that might give you the upper hand.
• Leading versus lagging metrics. Lagging metrics explain the past, and leading metrics give you a predictive understanding of the future; the latter are, consequently, better because they allow you to act upon them.
• Correlated versus causal metrics. In correlated relationships, two metrics change together, and one produces changes in the other. If you find a causal relationship between something you can control (ads) and something you want (revenue), then you can change your future.

The One Metric That Matters (OMTM)

“One of the keys to startup success is achieving real focus and having the discipline to maintain it,” write Croll and Yoskovitz. And, in analytics, achieving focus is all about picking a single metric and caring about it above all else. Of course, this doesn’t mean that there’s only one metric every founder of every possible type of startup should be concerned about while disregarding all others; but it does mean that, at any given stage, for a certain type of startup, there is one – and only one – metric that you should take into account. Suitably, the authors refer to it as the One Metric That Matters – or OMTM, for short.

There are four reasons to exclude all metrics but the OMTM until you get to the final stage of your development:

• It answers the most important question you have. And the most important question, at any given time for your startup, lies in the riskiest areas of your business. Discover it and track the related metric – that’s the OMTM.
• It forces you to draw a line in the sand. Picking an OMTM implies devising a clear goal as well. The OMTM, as a rule, encompasses your most important objectives.
• It focuses the entire company. Instead of “data puking” – that is, boring your employees with thousands of numbers – displaying just your OMTM prominently (through web dashboards, on TV screens, or in regular emails) should translate to synchronized company focus.
• It inspires a culture of experimentation. The lean startup movement is all about experimentation and moving through the build→measure→learn cycle as quickly and as frequently as possible. However, disorganized experimentation might lead to “big-F Failures.” On the contrary, across-company experimentation rallied around a single metric (the OMTM) inevitably leads to better processes and products via “small-F failures.” “Failure that comes from planned, methodical testing,” write Yoskovitz and Croll, “is simply how you learn.” And the OMTM allows you to learn in relation to a predefined objective.

A great example of an OMTM coming from the restaurant industry is the ratio of staff costs to gross revenues for the previous day. When the ratio is bigger than 1:3, then you’re either spending too much on staff or not deriving enough revenue per customer.

The five stages of a startup

Now, this ratio should only interest “Revenue stage” restaurants. However, a startup usually goes through four other stages before it can even deal with OMTMs of this sort.

• Stage One: Empathy. In the first stage, you’re trying to identify an unmet need shared by many people. Your goal is to devise an idea that should meet it in a way these people will pay money for at scale. 
• Stage Two: Stickiness. Once you formulate the right idea and find evidence for interest from a sizable market, it’s time to get sticky and search for the right mix of products or features that might keep your potential users around. The key at this stage is engagement, which you can measure by the time users spend interacting with you or the rate at which people return. Be willing to iterate on anything at this stage, while striving to figure out the simplest, lowest-friction path between your user and the “aha” moment you’re trying to deliver.
• Stage Three: Virality. Virality refers to the spread from your existing, “infected” users to new users. So, it’s not about buying ads, but about organic growth from within the community of your current users. There are three types of viral growth: inherent (e.g., via collaborating on Google Docs), artificial (via gamification or incentives), and word-of-mouth virality. In addition to the viral coefficient (the number of new users per existent users), another metric of importance at this stage is the viral cycle time, the amount of time it takes for a current user to invite others.
• Stage Four: Revenue. At this stage, your goal is pretty tangible: a sustainable business with healthy margins. Because of this, your metrics are bound to shift from usage patterns to business ratios. The core equation for the Revenue stage is the money a customer brings in minus the cost of acquiring that customer (ROI). However, it’s also very important to constantly realign your OMTM and figure out where to focus: more revenue per customer, more customers, more efficiencies, greater frequency, and so on. 
• Stage Five: Scale. Once revenues and margins are within your desired targets (as set out in your business model), it’s time to grow. You can scale only after you know your product and your market intimately. Whereas up to this stage things like compensation, API traffic, channel relationships, and competitors might have been a distraction, now they are of utmost interest. Because you’ll need to have more than one metric at a time at the scale stage, it’s important to set “a hierarchy of metrics that keeps the strategy, the tactics, and the implementation aligned with a consistent set of goals.”

Final Notes

Packed with memorable case-studies and loads of actionable advice, Lean Analytics works as a sort of encyclopedia on the second element of the build→measure→learn lean development cycle.

So, if you want to build a better startup – and you want to do it as fast as possible – then Lean Analytics might be the book for you.

12min Tip

Don’t sell what you can make; make what you can sell. And to discover what others are willing to buy, discover your One Metric That Matters and disregard all others.